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ABSTRACT. 


To date, little progress has been made toward the establishment 
of scientific procedures by which a military manager can evaluate 
complex cybernetic systems suchas management information systems. 

To provide a methodology yielding quantitative results which may 
assist a commander and his staff inthis analysis, it is proposed that 
management information systems be evaluated.as a whole by a tech- 
nique defined as the semantic differential. Each characteristic of the 
system evaluated is compared to a standard or reference characteristic 
desired in a well designed system and a value is assigned based on 
the closeness of the comparison. 

A second, more detailed, approach is also considered. This 
technique detects redundancy and lack of responsiveness ina system 
by means of a matrix model of the system. Using input-output analysis 
on the model, it is possible to determine excessive routing of informa- 


tion and lack of the necessary information to compute decision rules. 
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I. INTRODUCTION 


A. BACKGROUND AND PURPOSE 

Every military resource manager is a decision maker anda 
problem solver. For example, the squad leader is faced with the task 
of managing his men, their equipment, and available time in a mission 
oriented environment. The description of the task confronting a com- 
manding general of anarmy is similar; the only difference being that 
of proportion. Each of these men accomplishes his task in generally 
the same manner. Each gathers data, converts the data into meaningful 
information, and then uses the information to make decisions. 

In an organization with a hierarchal structure suchas the Army, 
information plays a second and equally important role. Each manager 
at every level of command is required to collect certain items of in- 
formation and report these items to higher levels of command. This 
information is then used by the managers at these higher levels of 
command to make their decisions. 

If the systems used by the managers at each echelon of command 
to gather and analyze information are not adequate, there is a high 
probability that the information used will be inaccurate, incomplete 
and/or irrelevant. In this case, not only does the performance of 
the manager suffer, but also higher levels of command receive ''bad'! 


information which can and usually does result in untimely or incorrect 





management decisions. This frequently leads to the imposition of 
management information and data requirements on lower levels that are 
far in excess of what can reasonably be useful to higher echelon manage- 
ment. Managers, therefore, continually en to improve their means 
of gathering data, of molding it into useful information, and of analyzing 
this information to suit their needs. 

The problem confronting managers, and the on this paper 
attacks is "how cana complex system such as a management informa- 
tion system (MIS), with as many definitions as ewe are systems, be 
analyzed and evaluated? '' "Is there a methodology for evaluating an 
existing management information system?" "If someone should design 
a MIS for the Army which they claim is better than any existing MIS, 
how can this be verified? " 

The problem of analyzing and evaluating a MIS is extremely dif- 
ficult. A few important factors contribute to this. First of all, when 
one considers a large complex system such as a MIS for a private 
industry ora logistical MIS for the Army, one must deal in the 
abstract. This is due to the fact that it is impossible to accurately 
define the components of the system, it is difficult to measure how the 
system will function in a given environment, and the human factor's 
influence on the overall operation of the system is not readily measure- 
able. Secondly, the characteristics of a MIS that may be rated as 
excellent for one organization may prove ineffective for another. This 


is because a MIS must be tailored to meet the demands of its user. 





Hence, it is necessary to describe the characteristics of a MIS in the 
most general terms, as the reader will soon discover. The last 
obstacle to analyzing a MIS is that there does not seem to exist a set 
of standards against which to measure a particular MIS. Due to the 
abstract nature of the system, quantifiable measures are not readily 
identifiable. 

This thesis represents a beginning step toward satisfying a need 
in the Army today. There has been a great deal of research conducted 
in the areas of information theory and Eater techniques, but. 
almost all effort has been directed toward such technical subjects as 
file organization, storage and retrieval procedures, coding and de- 
coding, and transmitting procedures. If the eeadien is interested in 
the more technical aspects of information systems evaluation, he is 
referred to Refs. 2, 7, 9, 11, 15, 18, 40, 48, 56, 62, and 69. A 
critical review of the literature has not revealed very many techniques 
that a middle or high level manager or military commander can use to 
evaluate the MIS he is currently using or a proposed MIS intended to 
replace it. In short, the military manager has been left to use his 
intuition and past experience (some refer to this as military judgement) 
to satisfy his needs in this area. This paper presents two operations 
research techniques that he can use to scientifically evaluate a complex 


system such as a MIS and suggests that many more procedures exist. 











B. DEFINITION OF TERMS 
1. Management Information System 
The introduction of the computer as an important tool of 
management caused emphasis to be placed on the deliberate design and 
implementation of information systems. It seems strange that to date, 
there does not exist a generally agreed upon definition of a management 
information system. This fact adds to the problem of evaluating such 
a system for it is extremely difficult to evaluate an ill-defined subject. 
The term, management information system, has been used to describe 
a multitude of electronic data processing equipment or devices, data 
collection systems, and even clerical arrangements. This paper will 
not use the term in these contexts. 
This paper will use the term as it is defined by the Office of 

the Chief of Staff, U. S. Army: 

"Management Information System: An organized 

assemblage of resources and procedures required 

to collect, process, and disseminate data for the 

purpose of converting it to meaningful information 

for decision making in executing the command/ 

management functions of planning, organizing, 

directing, coordinating, and controlling the use 

of resources to accomplish missions and tasks.’ 

[Ref. 47] 
Terms synonomous with MIS are, Management Control Systems, and 


Command and Control Systems. This paper will also use the term, 


system, to mean MIS. 
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Data are facts or inputs to the system which can take ona 
variety of forms. They are the reports or image of the units and 
activities of the Army which are collected and stored. Data are the 
elements of which information is comprised. [Ref. 12] 

3. Information 

Information is the system output which results from the con- 
version of data into a product which enables a manager to take action 
within a given frame of reference. [Ref. 14] 

4. Operations Research Technique 

An operations research technique is a procedure which 
structures a real life situation into a quantitative model, abstracting 
the essential elements so that a solution relevant to the decision 
maker's objectives can be sought. The technique allows a scientific 
approach to decision making. [Ref. 33] 

5. Decision Factor 

A variable, the value of which, when compared against an 

arbitrary rule or standard, determines a solution relevant to the 


decision maker's objectives. 


ee Oo COPE 
It is this author's contention that the techniques of operations 
research lend themselves to the analysis and evaluation of a manage- 


ment information system. It is further contended that there exists 
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relatively simple techniques that military managers, or any user of 
the system, could use without intensive special training. 

This paper will discuss two general approaches to the problem of 
the analysis and evaluation of management information systems. The 
first approach is to consider the system as a whole, qualitatively 
evaluate it with respect to the characteristics of a well-designed MIS, 
and then convert these qualitative measurements to quantitative 
measurements in order to allow comparison with other systems. One 
advantage of this approach is that it identifies both the weak points and 
the strong points of the system. The manager can then retain the 
strong points while strengthening the weak points, thereby improving 
his system. Secondly, the approach is so simple that an untrained 
analyst can implement the approach. 

The second approach would be used to amplify the results of the 
first more general approach. For example, if the first technique 
indicated redundancy in the system, the second technique could 
measure how much redundancy exists. The second approach uses 
the many operation research techniques available to measure specific 
aspects or characteristics of a system. This paper will consider in 
detail only one such technique, a technique which a manager could 
possibly use without the help of an operations research analyst. 

Other techniques will be mentioned and briefly discussed with relation 


to the evaluation of a MIS. 
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Both approaches will be applied in this paper to partially evaluate 
a MIS recommended by a civilian contractor to replace many of the 
systems now inuse inthe Army. This system, an integrated materiel 
readiness, supply and maintenance MIS, is currently awaiting imple- 
mentation by the Army and is expected to replace three separate 
categories of management information systems. 

It should be noted that the techniques discussed in this paper are 
not limited to the evaluation and analysis of a MIS ata certain level of 
management operating under a particular cae The techniques are 
equally applicable to the logistics battalion utilizing a manual MIS as 


they are to a computerized Army-wide command information system. 
TCG MEDS, SSS OD IM VIE SiO pate! 


A. GENERAL COMMENTS 

This portion of the paper will present a technique for evaluating 
any complex system which does not lend itself to quantification. The 
technique was originally used to evaluate or measure in quantifiable 
terms what meaning a concept might have to a group of people. The 
reader, upon completion of this section, will probably recognize that 
the personnel efficiency report and several types of inspection work- 
sheets use a similar technique. Management information systems can 
lend themselves readily to this type evaluation. If properly utilized, 
this procedure will enable any user of a MIS to evaluate his current 


system and a proposed replacement for it, and then compare the results. 
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The next section of the paper is comprised of a description of a 
technique called the semantic differential. Following this will bea 
discussion of the characteristics of a well-designed MIS. The last 
section of this portion of the paper will apply these cha nen to 
the semantic differential and will show the evaluation of a proposed 


MIS. 


B. THESEMANTIC DIFFERENTIAL 

The semantic differential is a combination of controlled association 
and scaling procedures. The evaluator is provided with the charac- 
teristics of a well-designed system to be differentiated and a set of 
bipolar adjectival scales against which to doit. His only task is to 
indicate, for each item (pairing of a characteristic witha scale, ) the 
direction of his association and its intensity ona n-step scale. [Ref. 30] 

Figure 1 is an illustration of the semantic differential technique 
used to evaluate a complex system. Assume for illustrative purposes 
that a senior commander has asked ten of his staff officers and sub- 
ordinate commanders to evaluate the system in question. 

The choice of the characteristics used is entirely up to the manager 
and/or analyst conducting the survey. Obviously, the more a person 
knows about the system being evaluated, the more meaningful the 
characteristics will be, and the better the technique will work. The 
technique also permits the military manager to use military judgement 


factors and, in fact, encourages their use. 
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The factor weights are assigned by the manager based on his 
judgement of the relative importance of the characteristics listed with 
respect to his management objectives. In the example outlined in 
Figure 1, the manager believed that ee one and five were 
more important than the rest. This procedure is defensible in the 
absence of an external reliable criterion. [Ref. 30] 

Once the decision maker has completed his survey, he can evaluate 
his system intwo ways. First, each characteristic may be evaluated 
separately. Inthis manner, the strong and seek areas of the system 
can be identified. For example, in Figure l, by adding up the checks 
fem characteristic one, one can determine a rating of 10 which, on an 
ordinal scale, would indicate a strong point. Characteristic 3, on the 
other hand, has a rating of -14; obviously a weak point in the system 
that would require corrective action. 

The second method assigns a scalar number to the entire informa- 
tion system. The procedure is as follows: 


system rating 


tr 
o 
bd 
| 


a) = (SieOMOrscoluiManeipe) — 552.010, = lec oS 


Wes = Sfactomweieht for characteristic i, 1=1,2,.....m 


h 


= number of checks received by the ;° column 


of characteristic 1. 


The system rating, R, is then equal to sum of the column 
totals: 
R = 2S . 
j (3), 
where = (8) = j 2 Nj jW; ; fee ey Oy ol, Oe enlistees 
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For the example in Figure I: 


(43) =e 4e 4o) eal is: 


S(42) = 2[(4x3) + (1x1) + (3x1) + (2x3)] = 44 


S041) = 1[(1x3) + (5x2) + (1x1) + (1x3)] = 17 
S = 
( 0) 
S = _1{(2x2) + (3x1) + (3x1) + (2x3)] = -16 


(-1) 
S(_2) = -2[(1x3) + (1x2) + (5x1) + (2523 aes 6 


S(_3) = =3(tisesimeriGls1)} = -12 
R = 184+44+17+0-16-36-12 = 15 

The decision maker now knows that, based on the sample of the 
population which he surveyed, his MIS has a scalar rating of 15 which 
indicates only that the system appears to have more strong points than 
weak points. By comparing the score obtained by the system against 
the ratings of the other systems derived by the same raters, the 
manager can determine a relative overall effectiveness index of the 
system. 

The scalar rating taken by itself may be misleading. For example, 
a system may have a large positive rating and still have one or more 
characteristics rated very low by the majority of raters. Thus, a 
system selected because of a high R rating may in fact be sorely 
lacking ina few important areas. Hence, the decision maker should 
also evaluate the ratings received by each separate characteristic. 
By doing so, he can identify specifically the strong and weak areas of 


his system. 
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Thus, the manager who considers the R value of the system together 
with the dispersion of evaluations for each characteristic will be able 
to determine the strong and weak points of a MIS and also compare it 
to other systems. Furthermore, by the identification of the weak 
points in his system, the manager can initiate the next step in the 
systems design cycle by taking necessary actions to correct those 
weak points. 

In order to use the technique of the semantic differential it is 
necessary to determine the characteristics of a wae designed MIS. 
The next section is an examination of these characteristics. 

C. CHARACTERISTICS OF A WELL DESIGNED MANAGEMENT 

INFORMATION SYSTEM 

1. General 

The research by the author in this area was done by a critical 
review of books, journals, and technical reports relating to the design 
of management information systems. Every authority appeared to have 
his own ideas about what characteristics a MIS should have. The 
characteristics discussed here are those that are accepted by most 
and, in the author's opinion, are most applicable to the military 
Organization. It should be noted that there are no rules or standards 
in existence which say that the characteristics discussed in this paper 
are ''good" or all-inclusive. It is believed that this derived set of 
characteristics are most important in that they provide a framework 


for the evaluation of a system regardless of the evaluation technique 
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GENERAL 
Data Base 


Information 


Type of System 


Long-short Range Perspectives 


Flexibility 


Responsiveness of the System 


Capacity 


Acceptability 


SPECIFIC 

a. Useful to all levels of manage- 
ment in the organization. 

b. Specialized data bases held to 
a minimum. 


a. Accurate. 

b. Complete and comprehensive. 

c. Available and relevant to all. 

d. Understandable, meaningful 
and readily usable. 

e. Timely. 

f. Compatible to all requirements. 

2g Information is of high quality. 


a. Justified by structure and 
eq Win elaemts.. 
b. Compatible with other systems. 


a. Provides basis for decisions 
in policy and procedures. 

b. MIS measures over-all effec- 
tiveness. 

c. MIS establishes trends, permits 
management by exception. 


a. Capable of being updated with 
minimum down-time and costs. 

b. Can adapt to changes in doc- 
trine and operational reqmnts. 


a. Responsive to the demands of 
the commander. 

b. Responsive to the needs of 
subordinates. 

c. Maximum automatic control is 
mermirtrea. 

d. Each person given all informa- 
tion needed. 


a. Operates under maximum work 
load conditions. 

b. Information capacity of any 
channel is not exceeded. 


a. Allusers accept system. 
b. System is simple and reliable. 





TABLE I. Summary of MIS Characteristics 
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GENERAL | SPECIFIC 


Redundancy a. Data not reported by many 
channels. 
b. Direct channels of information 
flow. 


TABLE I. (con't) Summary of MIS Characteristics 
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used. A summary of the characteristics to be discussed in this paper 
is listed in Table I. 
2. Data Base Characteristics 

A management hierarchy exists in the military with different 
information needs at its various levels. The question arises as to 
how many data bases are needed in the MIS to serve the various levels. 

The bottom section of Figure 2 reflects all of the necessary 
details of each logistical operation in a military service. Operating 
management does not need all this detail, but it does need a subs et of 
i, as indicated by the center section of Figure c. At the top, policy 
management needs yet another subset of the overall data base. Can 
these varying needs be served by a single data base, or will it be 


necessary to structure three different data bases? [Ref. 31] 


FIGURE 2. How Many Levels of Data Base? [Ref. 31] 
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A similar problem, but one having a vertical rather thana 


horizontal alignment, is shown in Figure 3. Can maintenance manage- 


ment use the same data base as the supply or personnel people, or is 


cal 





each intent on having a data base of it's own? Inthe Army today, there 
is a tendency to follow the latter direction. A major challenge then to 
the designer of a MIS lies in trying to integrate a data base so that 

it can be useful to all organizational levels and components. The 
objective is neither centralization nor decentralization of management, 


but rather to provide a system that will accommodate either one. 


FIGURE 3. How Many Specialized Data Bases? [Ref. 31] 
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There are opponents to the concept of complete data bases 


for integrated information systems. These people believe in the theory 
of the concept, but also believe that any attempt to design such a sys- 
tem is doomed to failure simply because of the varied and unpredictable 
nature of the data required for managerial planning. Since the data 
required depends upon the problem being studied and these problems 


cannot be foreseen, they feel the integrated system is hopeless. 


Za 








Advocates of the integrated MIS hold the view that current 
data base design concepts which facilitate the retrieval of data elements 
in response to unstructured and non-predetermined management requests 
contribute significantly to the realization of the integrated system. 
These people believe that non-integrated systems have many dis- 
advantages. First, duplication of information coverage exists as each 
level and component uses and interprets information to suit their own 
needs. Secondly, a breakdown in communications is apt to occur as 
identical information is distorted by bias and atic in the organiza- 
tion. [Ref. 19] Lastly, confusion tends to result as policy formulation 
and operating decisions are complicated by the need to evaluate con- 
flicting information supplied by different data bases. [Ref. 5] Inthe 
final analysis, the degree of integration of the data base is a function 
of the Soets involved to obtain this integration. Thus, although a com- 
pletely integrated data base might be desired, the costs incurred may 
make a less integrated system more realistic. The current state of 
integrated data bases in the Army is well described in Refs. 6, 45-47. 

In summary, one characteristic of a well designed MIS is 
that its data base be so structured that it is useful to all levels and 
components of an organization; the goal being a completely integrated 
data base. This characteristic easily lends itself to the semantic 
differential, particularly if all levels and components of an organiza- 


F10On are represented in the evaluator’s survey. 
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3. Information Characteristics 

It can be argued that any MIS is only as good as the informa- 
tion that flows through it; hence, information must be evaluated. It is 
necessary, therefore, to determine a set of characteristics which may 
be applied to the semantic differential. One study interested in this 
area, Ref. 17, researched all available relevant books, reports, and 
papers and prepared a set of seven characteristics of information as 
listed below: 

a. Accuracy (precision and validity) 

b. Quantity (completeness, comprehensiveness, and pre- 
dictability ) 

c. Relevance (pertinence) 

d. Simplicity (understandable, meaningful, and readily 
usable) 

e. Timeliness (accessability) 

fi. Compatibility (standardization) 

g. Quality Rene and worth) 

It should be noted that each element in the set of information 
characteristics lends itself to detailed analysis by advanced operations 
research techniques. For example, a recent study by the Logistics 
Research Project of George Washington University attempted to relate 
the quantity of information submitted by Navy maintenance units to the 
accuracy of the data comprising that information. In this case, the 


volume of information could be measured, but its accuracy could not. 
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By utilizing standard multiple regression analysis, cluster analysis, 
and discriminant analysis techniques, it was shown, for particular 
Naval units, that there existed a positive relation between the volume 
of information submitted and the accuracy of the data reported. Hence, 
by measuring the quantity of information in the system, an analysis of 
the accuracy could be made. [Ref. 28] This example points out a use 
of the semantic differential not heretofore discussed. The items or 
characteristics making up the differential may be used as starting 
points for more advanced detailed analysis. 
4. Type of aon Manual or Computer-based 

The type and complexity of any management information system 
must be determined by the decision maker in terms of his own organiza- 
tional requirements and structure. [| Ref. 42] The Army Materiel 
Command, an agency responsible for most of the materiel in the Army, 
would find great difficulty in managing their assets if they were to rely 
on a manual system. On the other hand, an ammunition company 
commander utilizing 2 computer-based MIS for his internal mana ge- 
ment would probably tend to become so completely tied up in computer 
operations that he would neglect his primary duty of leadership to his 
men. 

In addition to analyzing his organizational structure, the 
manager must also consider the information that is to be processed. 
For example, if he processes large amounts of information, has many 


repetitive operations, has a number of interacting variables, requires 
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accuracy and speed, then a computer could certainly be used to best 
advantage. [Ref. 21] 

This characteristic is closely related to the cost of a com- 
puterized system ata given instant in time. An organization may well 
require a computer but not be able to afford one. However, shoulda 
break-through in computer technology occur which would reduce the 
costs to the user, the same organization might find it suddenly feasible 
to convert from a manual system to a computer-based system. 

The manager must also look outward - determining the type 
of system he desires to use. This author is convinced that sometime 
in the relatively near future a worldwide military management informa- 
tion system will be utilized by all services. At present, most of the 
major service commands, the different unified and specified commands 
and their subordinate commands each have their own MIS. Each of 
these commands, while having different missions and objectives, have 
many common data bases and software requirements. Compatibility 
between the systems of the commands would permit a eae capa bil- 
ity at a lower cost than would be possible if systems were totally 
different. Compatibility refers to the situation where the hardware and 
the software at one level ina hierarchy is able to work with the hard- 
ware and software at other levels. Thus, in determining what type of 
system to implement, the manager should always consider compat- 


ibility with the systems of his higher (and lower and equal) commands. 
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5. Long-range and Short-range Perspectives 

In order to be effective as a timely and efficient management 
tool, a management information system must view organizational 
resources and functions from long-range and short-range perspectives. 
As a long-range management tool, it must provide a basis for decisions 
in the areas of policy, doctrine, and procedures. As a short-range tool, 
it must measure over-all effectiveness, be capable of establishing and 
displaying trends and provide means for the detection and avoidance of 
incipient problems by the technique of cane fern by exception. [Ref. 4] 

6. Responsiveness of the System 

Any well designed MIS should be responsive to management. 
This means that the system must have the ability to process pertinent 
information at all levels of an organization and cause implementation of 
decisions ina timely manner. This timely information must be able to 
meet dynamic and fluid command and control requirements. [Ref. 5] 

Responsiveness can be achieved in at least two ways. First, 
the system must be responsive to the demand of the commander. A 
system that has relevant and accurate information flowing in it is use- 
less to the manager if he cannot receive this information before decisions 
are required. The importance of the system's ability to deliver the 
required information to the decision maker when he needs it cannot be 
understated. 

Responsiveness aiso works in the opposite direction - from 


commander to subordinate. Once the commander receives the required 
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information and makes his decision, the system should be able to issue 
orders based on that decision to subordinates within a certain time; the 
time differing with different type decisions. 

The MIS should be designed so as to permit maximum auto- 
matic management control. This implies relegating to machines all 
mechanical decisions, leaving for consideration only those problems 
requiring judgement and more difficult analysis. This type responsive- 
ness would produce a more accurate, more consistent, and generally 
better work product. It would also permit more time for management 
to devote to exceptions, creative thinking, basic planning, and self- 
improvement. [Ref. 24] 

7. ‘Flexibility of the System 

Flexibility is the ability of the system to adapt to unforeseen 
changes of a permanent nature in doctrine, policy, or operational 
requirements. A system should also be capable of evolutionary im- 
provements by taking advantage of state-of-the-art advances as they 
occur and all experience gained from changing situations. [Ref. 17] 

All too often, though not universally, the manager 

is confronted with report formats that he may have 

agreed to, reluctantly, two or three years earlier. 

He has changed his requirements in evolutionary 

patterns and will continue to change as his grasp of the 

job improves or as the area he controls undergoes 

change. He cannot use what the computer now spews 

out in great quantity at frequent intervals, and his 

requests for different reports are met by shocked 

references to reprogramming costs or by promises 


that the changes can surely be made within the next 
year if all goes well. [Ref. 28] 
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Along the same line, a MIS designed for the U. S. Army Pacific 
Command, for example, should be capable of adapting to the various 
systems and doctrine passed down from the Department of the Army 
such as a change in the army maintenance records system. 

8. Capacity 

Capacity is a measure of the ability of a system to operate 
effectively under maximum workload conditions. Regardless of whether 
a MIS is manual or computer-based, if the capacity is exceeded, little 
opportunity will be available for study or maton of information pekone 
decisions are made. Indeed, decisions may be made without supporting 
information. The non-technically qualified user of a system would 
probably find it difficult to accurately evaluate this characteristic. 
Using the Semantic Differential, he could only record his feelings, in- 
tuition, or judgement about capacity and this rating may in fact differ 
from the actual case. The decision maker could note however, how 
many people ''felt'' the capacity of a system was inadequate, for example, 
and investigate this characteristic more closely. 

9. Acceptability 

Acceptability is an extremely abstract characteristic of a MIS, 
yet it lends itself most readily to evaluation by means of the semantic 
differential. It is defined as a measure of a system's ability to develop 
a positive user's attitude toward the system and it's objectives. A 
system, in order to be effective, must be accepted by the people who 


use it. Inthe case of a new or proposed system which has yet to be 
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used, the differential would measure potential users' initial impressions 
of how effective they think the system will be, based on their interpreta- 
tion of the system proposals. 

The keynote to user acceptance and use is simplicity. The 
more rules and restrictions involved in system operation, the better the 
chance for error. A simple system will enhance the learning process 
and will make the overall operation more efficient. 

Reliability is also a key factor in user acceptance. Unexpected 
or irregular performance must be held toa leone Many potencies 
supporters of a system have been lost because of embarrassing and 
unexpected system performance. [Ref. 28] 

10. Redundancy 

Redundancy is an undesirable characteristic of a MIS in that it 
implies unnecessary duplication. It is important that the reader under- 
stand that some duplication may be necessary if, for example, the 
designer desired a cross check of data for error detection purposes. 

If redundance in data serves no purpose, it may result ina waste of 
man-hours throughout the organization served by the system. Further- 
more, it tends to throw confusion into the act of decision making by 
presenting multiple interpretations of data, as the manager receives 
seemingly conflicting information. 

Conflicting information is due to distortion or bias which 
appears as information is transmitted through an organization. Bias 


occurs as information is eliminated, modified, or added before a 
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message is relayed to the decision maker. [Ref. 2] Thus if identical 
data elements are originally reported by two sources, there results 

two possibly unequal interpretations of these data by the time they reach 
| the decision maker. 

The reader should note that bias in a system is inversely 
related to the degree of automatic decision making discussed previously. 
The more a system is automated, the less chance there is for bias to 
enter the system. It should also be noted, however, that even a fully 
automatic system may be redundant simply because of faulty eaters 

In summary, one means of minimizing bias and distortion is 
to minimize redundancy in the system. Another means is to make the 
channels of information flow more direct to the appropriate manager, 
i.e., make the system more automatic. Redundancy of a proposed MIS 
for the Army will be partially evaluated in the second portion of this 


thesis. 


Peele SEMANTIC DIFFERENTIAL USED TO EVALUATE A MIS 

An example of a complete semantic differential that could be used 
to evaluate a MIS is shown in Figure 4. In this instance, factor weights 
have not been provided, therefore let We =o cere. , tie pve acaad.= 
acteristics discussed in the preceding section have been brought forward 
and have been applied to the semantic differential. 

The reader should bear in mind that this is only an example. The 


evaluator determines exactly what characteristics of his system he 
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he desires to have evaluated. Fora particular MIS, it is possible that 
the characteristics would be in much more detail than the general terms 
used in the example. 

As is shownin Figure 4, the technique has been applied to evaluate 
a proposed MIS recommended by a civilian research company. The 
evaluation was accomplished by comparing the characteristics of the 
proposed MIS (as reported in the project paper) to the characteristics 
of a well designed MIS. By using an adequate sample of the population 
and other standard statistical procedures, an eveluation could deter- 
mine the strong and weak points of a MIS and its relative standing to 
other management information systems. The technique also serves 
as the first step of a detailed analysis by breaking the large problem 


into a set of small, easier to handle, problems. 
III. DETAILED ANALYSIS OF INFORMATION FLOW 


A. GENERAL COMMENTS 

The last section dealt with analysis of a MIS when considering the 
characteristics of the system as a whole. This section discusses a 
methodology of analyzing a specific characteristic of the system. Thus, 
more precise conclusions can be reached with respect to the evaluation 
of the characteristic being considered. 

Every decision made in an organization is a function of the informa- 
tion the decision maker has at hand at the time of the decision. Ifa 
decision maker is lacking pertinent, relevant information, his decisions 


will reflect this. 
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It appears then, that there is a definite need for a technique which 
could analyze the information flow within an-organization. Such a tech- 
nique exists in the form of a simple matrix relation which identifies and 
traces data elements from their inclusion in source documents to the 
consolidated reports used by decision makers at various levels in the 
Organization. Comparison of this final derived matrix with a reference 
matrix constructed with prior knowledge of the data elements needed 
for each decision rule enables the analyst to indicate excessive routing 
of information or the lack of relevant information in a system. i other 
words, redundancy and certain elements of responsiveness as discussed 
in the preceding section are analyzed in detail. 

Homer [Ref. 34] improved upon this technique which was initially 
developed by Lieberman [Ref. 44]. The following is a description of 
Homer's model, slightly modified to apply to military management 


information systems. 


B. THE MODEL 

The model consists of a single matrix, M, which represents the 
entire management information system with respect to data elements, 
reports, and decision rules. The steps for forming the system matrix 
are as follows: 

Step 1: A row is established for each data element and report 

in the system. 
Step 2: A column is established for each report and decision factor 


in the system. 
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Step 3: The number 1 will be inserted in each cell to represent 
an item of data appearing ina report, one report used to 
produce another, or a report upon which a decision factor 
is based. An example of a decision factor might be man- 
days behind schedule. This factor, should it become too 
large would cause the manager to add more men toa shop 
Or take Other Corrective action. 

Step 4: Each report is represented by both a column and a row. 
Into each cell formed by the intersection of an ientheal 
row and column, the value -1 will be inserted. 

Step 5: All other cells will be labeled zero. 

An initial or preliminary analysis of the system matrix can now 
be made. First, any column which contains only zeroes should be 
removed. This indicates that a decision factor is not relevant to the 
system being considered or that no information is being reported upon 
which to base the decision and that a possible revision of the system is 
meeaea, A report column containing only zeroes indicates a clerical 
error, 

Next, any row which contains only zeroes should be removed. This 
indicates that a data element is superfluous to the system being evaluated. 
Again, a row representing a report containing only zeroes indicates a 
elerical error. 

If a column contains -1 as the only non-zero entry, the component 


represented by that column is an input to the system and the column 
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should be removed. Ifa row contains -1 as the only non-zero entry, 
the component represented by that row is an output of the system and 
the row should be removed. 

Inputs and outputs of the system can now be identified. Any row 
not containing a -1 entry is aninput. Similarly, any column not con- 
taining a -1 entry is an output. An output is normally a decision factor, 
but it may be a report or evena data element as the reader will observe 
in the application that follows this discussion. The solution area of the 
matrix consists of those cells formed by the tee eb of ‘ree see 
and output columns. 

Once the solution area has been identified, the solution matrix, 
M*, to the system matrix is derived by performing elementary column 
operations only. The procedure is to perform the necessary elementary 
column operations to reduce to zero each entry in an output column other 
than those in the solution area. The resulting values in the solution area 
represent the number of ways in which each input "reaches"! each output. 
The solution matrix, M*, is then subtracted from the reference matrix 
mentioned earlier, giving an indication of excessive routing of informa- 
tion or the lack of needed information at a particular level within the 
organization. For a detailed discussion and mathematical proof of this 


procedure, the reader is referred to Reference 34. 


Ca A MILITARY APPLICATION 
In 1967, the Office of the Deputy Chief of Staff, Logistics, Depart- 


ment of the Army, contracted a civilian research firm to design an 
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integrated MIS. This MIS was to provide materiel readiness, supply, 
and maintenance information to Army managers at each echelon of 
command or logistics support level. The design did not include auto- 
matic data processing programming or specifications for data process-~ 
ing equipment. The design description did, however, permit the Army 
to proceed with programming and operating instructions without further 
systems development effort. 
Generally, the designers of the proposed MIS claimed that their 

system would accomplish the below listed haan anreieaenbss 

Simplify and reduce reporting data elements 

Consolidate major reporting systems 

Improve data reliability 

Improve responsiveness 

Improve evaluation of operating elements 

Simplify report formats 

Diagnose potential management deficiencies 

The modified Homer's matrix evaluation technique discussed above 

will be used to evaluate the results of the first two listed improvements. 
It is not this author's intention to rigidly analyze the proposed MIS. 
Such a task is infeasible due to the fact that the contractor's report is 
not specific enough to accurately define the relationships between source 
data elements, records and reports, and Secistee factors to be used. 
Instead, a small portion of the over-all system, maintenance perform- 


ance evaluation at the depot level, will be extracted and examined. 
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Table Il lists all the data elements required for depot level main- 
tenance performance evaluation, a matrix code assignment for each 
milement, and the reports that utilize each element, 

Table lllis a list of the ten reports required by the proposed sys- 
tem. All reports are independent from each other except Report rj, 
Backlog Analysis by FSN, and Report r7, Unwarranted Equipment 
Evacuations. Report r5 is needed to produce Report Ty: Lack of 
Reparables. Report r7 is used to produce Tg, Unwarranted Maintenance 
Evacuations-Detailed Listing. 

The decision factors assumed to be used in maintenance perform- 
ance evaluation are listed in Table 1V. Also listed is the code assigned 
to each factor and the reports that compute each factor. These decision 
factors were extracted from the exhibits of Beas maintenance reports 
listed in the contractor's project report. It is assumed that each of 
these computations is to be compared against some standard set by the 
manager, thus indicating to him satisfactory or unsatisfactory perform- 
ance and the reasons therefore. 

Utilizing Tables II-IV, the system matrix, M, can be constructed by 
following the five steps listed in the last section. Figure 5 is the result- 
ing system matrix. 

A preliminary analysis of the matrix indicates that reports rg and 
¥rjo are actually outputs of the system and the matrix rows corresponding 
to these reports are removed. The solution area is then identified. 
Figure 6 shows the system matrix after preliminary analysis and 


identification of the solution area. 
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Data Element 


Maintenance Organization Name 
Master Schedule Item- No. 
Master Schedule Item- FSN 
Master Schedule Item- Noun 
Work Stoppage Part- FSN 

Work Stoppage Part- Name 
Work Stoppage Part- Reqn No. 
Program Qnty Required- Parts 
Critical Qnty Required- Parts 
Work Stoppage Date 


Ma ster Schedule Item- Critical Qnty 
Quantity Scheduled 

Quantity Completed~ Per Month 
Cause For Delay 

Equip. Scheduled to be Turned In 
Average Man-hours Programmed 
Average Man-hours Expended 
Man-hours Expended Code 

Actual Man-hours Expended 
Command Code 

Qnty of Equipment Inspected 

No. of Unwarranted Equip. Evacs 
Est. Man-hours Required for Repair 
FSN 

Brief Nomenclature- FSN 

myvpe- Model- Series 

PCN 

Initial Schedule Date 

Part Shortage-FSN 
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Code 





ml 
122 
123 
124 
125 
126 
197 
138 
ing 


Used in Reports: 


TyT3,T4,T5sTEsTo,TyQ 
¥y5T2,T3,T4,T5,To,TyQ 
Ty>T2/T3,T4,Te,TQTl1Q 
¥j+T2,1T3,T4,Te,To,TyQ 
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Data Elements Required at the Depot Level 





Data Element Code Used in Reports: 


Part Shortage- Nomenclature i239 T1090 
Part Shortage- Quantity i234 rjo 
Requisition No. - Parts i132 Tio 
Equipment Shortage-~ FSN 1. Ig 
Equipment Shortage-~ Nomenclature 134 rg 
Quantity Short- Equipment ize Ig 
Document No. - Equipment 136 Ig 


No. of Direct Labor Men/Work Center 137 ry 


TABLE II. (con't) Data Elements Required at the Depot Level 
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Reports Required by Proposed MIS 


Backlog Analysis- By Work Center 

Backlog Analysis- By Federal Stock Number 
Work Stoppage Items 

Lack of Reparables 

Man-hour Utilization 


Assigned Time Analysis By Work Center 


Unwarranted Equipment Evacuations 
Unwarranted Maintenance Evacuations-~ Detailed Listing 
Equipment Shortages 


Parts Shortages 


f= Report used to produce another report. 


TABLE III, Reports Required at the Depot Level 
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Decision Rules of Proposed MIS 


No. of Equipments Behind Schedule 
% of Equipments Behind Schedule 


Man-days Behind Schedule 

Shop-days Behind Schedule 
Cumulative Programmed Output/Item 
Cause For Delay 

Cumulative Actual Output/Item 
Programmed:Actual Performance 
Critical Stoppages 

Assigned Time Index 


Rate % Unwarranted Equipment 
Evacuations 


Unwarranted Work Requests- 
Estimated Man hour 


Unit Evacuation Performance 


Code 
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Reports Utilized 
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TABLE IV. Decision Factors Utilized at the Depot Level 
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FIGUR& 5 The System Matrix 
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FIGURE 6 The Solution Area 
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The solution area is enclosed in the rectangle above. 


Note: 
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lhe Final Matrix 


FIGURE 7 
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Figure 7 shows the system matrix, M*, after the necessary column 
operations have reduced to zero each entry*in an eet ait column other 
than those in the solution area. Notice that some inputs reach the 
decision factors in more than one way. If, after subtracting M* from 
the reference matrix, there resulted negative values in the solution area, 
excessive routing of information might be indicated and revision of the 
reporting system would then be necessary. If positive values resulted 
in the solution area, it might indicate a lack of the information desired 
to compute a decision rule and again, a ee of the reporting system 
would be needed. 

For example, suppose that the decision maker desired that each 
input flow through the organization in such a manner as to reach its 
relevant decision factors from two different sources. A glance at 


Figure 7 shows that source data element i,, Master Schedule Item-No, 


2: 
progressed through the system and ultimately was utilized by decision 


factor d_, Cumulative Programmed Output/Item, from four separate 


5? 
reports. In this case, the system is redundant. Another glance at 
Figure 7, this time scanning row 15 3) indicates another need for a 


system revision. This time data element 1,2 needs to be added to other 


3 
reports as itis currently being utilized from only one report by each of 


its relevant decision factors. 


D. CONCLUSIONS 
The Homer model for analyzing the flow of information can bea 


valuable tool for the analyst for determining excessive routing of 
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information or the lack of required information. The technique is 
dependent upon the analyst's ability to identify all the individual com- 
ponents of an information system and the ability to establish a reference 
matrix against which to compare the derived solution matrix. 

The latter requirement provides the major obstacle to maximum 
utility of the model. It is a recognized difficult task to identify all the 
decision rules in a complex organization and it is even more difficult 
to identify the multitudinous parameters relating to these decision rules. 
Hence, the reference matrix may be, in aeneier, impossible to obtain. 
Even if this were so, the preliminary analysis made on the system 
matrix is extremely useful due to the fact that extraneous data elements 
and non-relevant decision factors are identified. 

The model provides several advantages to the analyst in addition to 
presenting a complex organization in relatively simple form. First, it 
becomes easy to divide systems analysis work amongst the members of 
an analysis group due to the nature of the system matrix. Secondly, 
the model lends itself very well to the computer. Another advantage is 
the experimentation that can be carried on with the model to determine 
the effects of system change without disturbing the actual system. 
Lastly, the informal communications system of an organization may be 
incorporated into the model, thus allowing the analyst to evaluate this 
important segment of an information system and compare it to the formal 


structure. [Ref. 37] 


49 








- 
; an 


IV. RECOMMENDATIONS 

It is anticipated that two separate groups of people will be able to 
use this thesis to their advantage -~- military managers and operations 
research analysts. This section of the paper offers recommendations | 
to these people as to how they may possibly use this thesis. 

There has been much effort to date to find ways to measure the 
technical aspects of management information systems. The overall 
measurement of the entire system from a manager's viewpoint has 
memactically been ignored. It is hoped that the military managér will 
find the techniques discussed in this thesis useful in measuring the 
effectiveness of his system. At the least, this thesis should alert 
military mangers that techniques are ivatianee and that future efforts 
to expand these techniques are worthwhile with respect to improving 
the processing of management information in the military. The author 
recommends that interested managers apply the techniques discussed 
in this paper to their existing MIS to determine if, in fact, the results 
support his intuition and military judgement of the ereterd or perhaps 
even identify weaknesses he had not previously considered. 

In viewing this thesis as a stage ina research continuum, there are 
several logical follow-on studies. First, this author has identified 
what he believes to be a set of characteristics common to all manage- 
ment information systems. It is recommended that these characteristics 
be tested and verified through empirical studies. The author suggests 


that the semantic differential could well be used as a technique to 
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accomplish this. By polling known experts in the field of MIS design 
and experienced users of MIS's, the differential would establish the 
acceptability of these characteristics. Secondly, the techniques them- 
selves should be verified. In other words, do the techniques result in | 
accurate measures of a system being considered? This is not an easy 
task. This author recommends that the Semantic Differential and 
Homer's input-output model should be applied to evaluate a system 
with known defects. If the techniques detect and measure these defects, 
it would indicate that they ice Finally, the author hopes that S 
discussing a few of the techniques available, other researchers will 
contribute more techniques, thus expanding the tools the manager can 


use to measure the effectiveness of his system. 


V. SUMMARY 

To date, there has been little consideration of the military manager's 
desire to analyze his management information system from a managerial 
viewpoint. Most of the evaluative techniques developed have been used to 
analyze the technical aspects of a system such eS file organization, coding 
and decoding, and retrieval and transmitting procedures. The military 
manager has been left to use his intuition and experience to satisfy his 
analysis needs. 

This thesis has been a first step toward satisfying the military 
manager's needs inthis area. It presented two operation research 


techniques that the manager could use to scientifically evaluate a MIS 
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and suggested that many more procedures exist. The first approach 
discussed, the semantic differential, considered the soetena as a whole; 
qualitatively evaluated it with respect to a set of characteristics common 
to well designed MIS's; and then converted these qualitative measure- 
ments to quantitative measurements to allow comparison with other 
systems. Thus, this technique was a combination of controlled associa- 
tion and sealing procedures which provided two separate analyzes: 

(a) the identification of strong and weak characteristics of a system 

and, (b) a scalar rating factor which, went B+cneeks used, allowed a 
comparison of similar systems. 

In order to use this technique, it was necessary to define the 
characteristics of a well designed MIS. After a rather comprehensive 
literature search, the author developed the below listed set of char- 
acteristics which could be used as a framework for the evaluation of a 
system regardless of the evaluation technique used. 

imeotructure of the data basic 

2. Characteristics of Information 

a. Accuracy 

b. Quantity 

Cou velevanee 

d. Simplicity 

e. Timeliness 

ib Compatibility 


g. Quality 
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3. Type of System-Manual or Computer-based? 

aye Ones and ohore-range Perspectives of the evetem 

5. Responsiveness 

6. Flexibility 

7. Acceptability 

8. Redundancy 
These characteristics served as standards against which to compare the 
characteristics of the MIS being evaluated. The strength of the semantic 
differential is a function of the quality of ree standards and the statis- 
tical procedures used in the evaluation. 

The second approach discussed was typical of the many techniques 
available which could be used to measure more definitively a specific 
characteristic of the system, e.g. a weak area identified by use of the 
semantic differential, This technique, an input-output model of an 
information system developed by E. D. Homer, traced the flow of 
information throughout the system without regard to an organizational 
structure. Asa result, excessive routing of information and the lack 
of required information upon which to compute decision rules were 
detected. In order to apply the technique, a manager needed only to 
know simple matrix operation procedures. The major disadvantage of 
this technique was that one assumed that information requirements and 
decision factors were known. Experience indicates that this is not 
always the case. The main advantage of this technique was that it 


enabled the analyst to model a complex system in simple form. 


Je. 





In conclusion, it has been demonstrated that there is a methodology 
for evaluating management information systems froma managerial 
viewpoint. Using operations research techniques, the military manager 
can combine his experience and judgement with scientific procedures to 
evaluate complex procedures. He can evaluate the system as a whole 
using a technique such as the semantic differential or he may choose to 
analyze a small portion of the system by using one of the many procedures 
known to the operations analyst today. The problem of analyzing un- 
quantifiable systems is formidable, but a solution is attainablé with 


operations research techniques. 
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